We consider the dynamic linear regression problem, where the predictor vector may vary with time. This problem can be modeled as a linear dynamical system, with non-constant observation operator, where the parameters that need to be learned are the variance of both the process noise and the observation noise. While variance estimation for dynamic regression is a natural problem, with a variety of applications, existing approaches to this problem either lack guarantees altogether, or only have asymptotic guarantees without explicit rates. In particular, existing literature does not provide any clues to the following fundamental question: In terms of data characteristics, what does the convergence rate depend on? In this paper we study the global system operator -- the operator that maps the noise vectors to the output. We obtain estimates on its spectrum, and as a result derive the first known variance estimators with finite sample complexity guarantees. The proposed bounds depend on the shape of a certain spectrum related to the system operator, and thus provide the first known explicit geometric parameter of the data that can be used to bound estimation errors. In addition, the results hold for arbitrary sub Gaussian distributions of noise terms. We evaluate the approach on synthetic and real-world benchmarks.

Transmission grid congestion increases as the electrification of various sectors requires transmitting more power. Topology control, through substation reconfiguration, can reduce congestion but its potential remains under-exploited in operations. A challenge is modeling the topology control problem to align well with the objectives and constraints of operators. Addressing this challenge, this paper investigates the application of multi-objective reinforcement learning (MORL) to integrate multiple conflicting objectives for power grid topology control. We develop a MORL approach using deep optimistic linear support (DOL) and multi-objective proximal policy optimization (MOPPO) to generate a set of Pareto-optimal policies that balance objectives such as minimizing line loading, topological deviation, and switching frequency. Initial case studies show that the MORL approach can provide valuable insights into objective trade-offs and improve Pareto front approximation compared to a random search baseline. The generated multi-objective RL policies are 30% more successful in preventing grid failure under contingencies and 20% more effective when training budget is reduced - compared to the common single objective RL policy.

We consider the dynamic linear regression problem, where the predictor vector may vary with time. This problem can be modeled as a linear dynamical system, with non-constant observation operator, where the parameters that need to be learned are the variance of both the process noise and the observation noise. While variance estimation for dynamic regression is a natural problem, with a variety of applications, existing approaches to this problem either lack guarantees altogether, or only have asymptotic guarantees without explicit rates. In particular, existing literature does not provide any clues to the following fundamental question: In terms of data characteristics, what does the convergence rate depend on? In this paper we study the global system operator -- the operator that maps the noise vectors to the output.

A plan to create a clean electricity system by 2030 promised by Labour before the election is "immensely challenging" but still "credible" if ministers take urgent action to fix Britain's sluggish planning system, the energy system operator has said. Britain could become a net exporter of green electricity by the end of the decade at no extra costs to the energy system under the plans and bills may even fall if ministers make the right policy changes, according to the operator. The newly formed National Energy System Operator (Neso) put forward the conclusions as part of its official advice to new ministers on how to reach Labour election pledge to decarbonise the power system by 2030. Fintan Slye, the chief executive of Neso, said: "There's no doubt that the challenges ahead on the journey to delivering clean power are great. However, if the scale of those challenges is matched with the bold, sustained actions that are outlined in this report, the benefits delivered could be even greater."

Understanding and predicting the electricity demand responses to prices are critical activities for system operators, retailers, and regulators. While conventional machine learning and time series analyses have been adequate for the routine demand patterns that have adapted only slowly over many years, the emergence of active consumers with flexible assets such as solar-plus-storage systems, and electric vehicles, introduces new challenges. These active consumers exhibit more complex consumption patterns, the drivers of which are often unobservable to the retailers and system operators. In practice, system operators and retailers can only monitor the net demand (metered at grid connection points), which reflects the overall energy consumption or production exchanged with the grid. As a result, all "behind-the-meter" activities-such as the use of flexibility-remain hidden from these entities. Such behind-the-meter behavior may be controlled by third party agents or incentivized by tariffs; in either case, the retailer's revenue and the system loads would be impacted by these activities behind the meter, but their details can only be inferred. We define the main components of net demand, as baseload, flexible, and self-generation, each having nonlinear responses to market price signals. As flexible demand response and self generation are increasing, this raises a pressing question of whether existing methods still perform well and, if not, whether there is an alternative way to understand and project the unobserved components of behavior. In response to this practical challenge, we evaluate the potential of a data-driven inverse optimization (IO) methodology. This approach characterizes decomposed consumption patterns without requiring direct observation of behind-the-meter behavior or device-level metering [...]

The US Department of Energy has recognized this trend, recently awarding $3 billion in grants to various "smart grid" projects that include AI-related initiatives. The excitement about AI in the energy sector is palpable. Some are already speculating about the possibility of a fully automated grid where, in theory, no humans would be needed to make everyday decisions. But that prospect remains far off; for now, the promise lies in the potential for AI to help humans, providing real-time insights for better grid management. Here are four of the ways that AI is already changing how grid operators do their work.

Generally, system operators conduct the economic operation of power systems in an open-loop predict-then-optimize process: the renewable energy source (RES) availability and system reserve requirements are first predicted; given the predictions, system operators solve optimization models such as unit commitment (UC) to determine the economical operation plans accordingly. However, such an open-loop process could essentially compromise the operation economics because its predictors myopically seek to improve the immediate statistical prediction errors instead of the ultimate operation cost. To this end, this paper presents a closed-loop predict-and-optimize framework, offering a prescriptive UC to improve the operation economics. First, a bilevel mixed-integer programming model is leveraged to train cost-oriented predictors tailored for optimal system operations: the upper level trains the RES and reserve predictors based on their induced operation cost; the lower level, with given predictions, mimics the system operation process and feeds the induced operation cost back to the upper level. Furthermore, the embeddability of the trained predictors grants a prescriptive UC model, which simultaneously provides RES-reserve predictions and UC decisions with enhanced operation economics. Finally, numerical case studies using real-world data illustrate the potential economic and practical advantages of prescriptive UC over deterministic, robust, and stochastic UC models.

Forecasting and decision-making are generally modeled as two sequential steps with no feedback, following an open-loop approach. In this paper, we present application-driven learning, a new closed-loop framework in which the processes of forecasting and decision-making are merged and co-optimized through a bilevel optimization problem. We present our methodology in a general format and prove that the solution converges to the best estimator in terms of the expected cost of the selected application. Then, we propose two solution methods: an exact method based on the KKT conditions of the second-level problem and a scalable heuristic approach suitable for decomposition methods. The proposed methodology is applied to the relevant problem of defining dynamic reserve requirements and conditional load forecasts, offering an alternative approach to current \emph{ad hoc} procedures implemented in industry practices. We benchmark our methodology with the standard sequential least-squares forecast and dispatch planning process. We apply the proposed methodology to an illustrative system and to a wide range of instances, from dozens of buses to large-scale realistic systems with thousands of buses. Our results show that the proposed methodology is scalable and yields consistently better performance than the standard open-loop approach.

At Microsoft, building a future that we can all thrive in is at the center of everything we do. On January 16, as part of the announcement that Microsoft will be carbon negative by 2030, we discussed how advances in human prosperity, as measured by GDP growth, are inextricably tied to the use of energy. Microsoft has committed to deploy $1 billion into a new climate innovation fund to accelerate the development of carbon reduction and removal technologies that will help us and the world become carbon negative. The Azure IoT team continues to invest in the platforms and tools that enable solution builders to deliver new energy solutions, customers to empower their workforce, optimize digital operations and build smart, connected, cities, vehicles, and buildings. Earlier, Microsoft committed $50 Million through Microsoft AI for Earth that provides technology, resources, and expertise into the hands of those working to solve our most complex global environmental challenges.

Just as power system operators are mastering data analytics to optimize hardware efficiencies, they are discovering how the complexities of artificial intelligence tools can do far more, and how to choose which to use. With deployment of advanced metering infrastructure (AMI) and smart sensor-equipped hardware, system operators are capturing unprecedented levels of data. Cloud computing and massive computational capabilities are allowing data analytics to make these investments pay off for customers. But it may take machine learning (ML) and artificial intelligence (AI) to address new power grid complexities. AI is a form of computer science that would make power system management fully autonomous in real time, researchers and private sector providers of power system services told Utility Dive.